Australian Hansard topic modelling analysis

pacman::p_load(stringr, tidyverse, scales, tidytable, deeptime, purrr, ggplot2, ggtext, glue)

params <- "k40_var0.05"
  
dtm_run_path <-  "coal_output/dtm/run_21Sep//all_2a_min_freq_150_21_09_21/"
  
docs_file <- "id_alliance_all_2a.csv"
  
  

source('scripts/report/coal_topics_FUNS.R')

paragraphs_df <- fread.(paste0("coal_data/04_model_inputs/preprocessed_datasets/", docs_file)) %>% 
  select.(-contains("lucy"))

Raw number of docs by party

relative to parliamentary activity

https://www.aph.gov.au/-/media/02_Parliamentary_Business/22_Chamber_Documents/224_Statistics/House_of_Representatives/Statistics_Historical/legislation_statistics.pdf?la=en&hash=6ADCACEE5832882C1E8660A06563DC9200F6EC54

https://www.aph.gov.au/Parliamentary_Business/Bills_Legislation/Bills_Lists/Details_page?blsId=legislation%2Fbillslst%2Fbillslst_c203aa1c-1876-41a8-bc76-1de328bdb726

dividing the counts of paragraphs discussing coal by the number of bills passed in parliament that year as a proxy for parliamentary activity. The results are then normalised so that 1 corresponds to the largest amount of discussion by any party, which corresponds to the Greens in 2019. Parliament has arguably become more active over time in part due to an increase in the number of MPs and Senators and in part due to the increased requirements of a larger population and a more complex economy.

With those differences taken into account, the periods between 1940 and 1960 and post-2005 stand out, corresponding respectively to

largest amount of discussion

years above the 90% value

1902, 1929, 1942, 1944, 1945, 1946, 1947, 1948, 1949, 1950, 1951, 1959, 1961, 2006, 2007, 2008, 2009, 2011, 2012, 2013, 2014, 2015, 2017, 2018, 2019, 2020

Bringing various elements together

filtering the minimal part of “other” in the period prior to 1975 and in general any topic before 1910 because the amounts of information are tiny and lead to wild fluctuations in proportion.

# From this article https://www.blueprintinstitute.org.au/polling_data
coal_mining_regions <- "Kalgoorlie|Parkes|Lyne|Hunter|calare|shortland|Flynn|Maranoa|gippsland|connor|capricornia"

all_topic_scores_df <- paragraphs_df %>% 
  select.(doc_id, electorate, final_alliance, year) %>% 
  left_join.(select.(topic_distributions, -c(V1, year))) %>% 
  pivot_longer.(c(`0`:sym(topic_nums)), names_to = "topic_no", values_to = "score") %>% 
  mutate.(topic_no = as.integer(topic_no)) %>% 
  left_join.(topic_labels) %>% 
  left_join.(select(elections, year, era, legislature)) %>% 
  filter.(year > minimum_display_year, 
          !(final_alliance == "Other" & year < 1975)) %>% 
  mutate.(label = paste0(str_pad(topic_no, 2, "left", "0"), "_", label),
          mining_region = str_detect(electorate, regex(coal_mining_regions, ignore_case = T)),
          labor_mining_region = final_alliance == "Labor" & mining_region == TRUE,
          country_mining_region = final_alliance == "Country" & mining_region == TRUE)

Foci of each party relative to average

(Replicating MH’s Fig. 10)

Differences range: -3.697844 6.038361 [1] “goldenrod3” “darkgreen” “firebrick” “blue”
[5] “black” “darkgoldenrod4” “darkred”
[1] “Country”
[2] “Greens”
[3] “Labor”
[4] “Liberal”
[5] “Other”
[6] “Country
coal” [7] “Labor
coal”

Manually aggregated categorys

TF-IDF ‘best categories’

best used to “confirm” that we’re not selecting the wrong thing.

By individual topic and era

00_transport01_national procurement02_electorate03_procedural statements04_carbon price05_Victorian brown coal06_coal supply07_QLD and WA resources08_elections & policy09_budget10_Committee on Coal11_environmental impacts12_interstate trade13_trade partners (UK, JPN, CHN)14_renewable energy projects15_export16_research17_welfare18_big business influence19_commonwealth vs states (JCB)20_water and farming21_arbitration22_employee entitlements23_excise and duties24_nuclear power25_procedural statements26_taxes27_coal miners28_commissions and committees29_other fuels30_employment31_production32_electricity generation33_Asian opportunities34_infrastructure35_bills and measures36_industrial relations37_iron and steel38_procedural statements39_climate change